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AWARDS ABSTRACT 


The invention related to an optical implementation of neural associa- 
tive memory systems, based on nonlinear, dynamical terminal attractors with 
adaptive threshold values for dynamic iteration of unipolar, binary neuron 
states . 


An object of the invention is to provide a Terminal-Attractor Based 
Neural Associative Memory (TABAM) system which, unlike the prior-art 
terminal attractor system of M. Zak, "Terminal Attractors in Neural 
Networks," Vol. 2, pp. 259-274, NPO-17832 (1989), is not defined by a 
continuous differential equation and therefore can be readily implemented 
optically with an inner-product neural associative n» en |ory of “ * J* Llu ' 
"Optical Inner Product Neural Associative Memory," NPO-18491, (1991). 


This objective is achieved in an optical neural network shown in FIG. 

10, where a set of N-tuple neuron-state vectors V={o", are 

stored as terminal attractors I if in which the iterative neural dynamics 

are given by 

- E T i&l x J ] +J i 
j-1 


where T L - is the ith row and jth column element of a weight matrix, and 
g(x L ) is 3 a sigmoidal threshold function, where 

I i = £>"’[g(x i )-u , ?] 1/3 exp(-p 1 ?[gr(x i )-u'?] 2 }. 

fll-1 



is a parameter that is the inner product between one of the stored vectors 
and a computed vector of a preceding iteration that is a similarity between 

one of the stored vectors and the computed vectors, P™ is a constant P™, 
and g(x j) is a sigmoidal function of tanh ax j, where 1 irv.gr (x.,) is a binary 
component • 
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Yet another object of this invention is to. provide a unipolar inner- 
product TABAM (UIT) and a crosstalk reduced inner-product TABAM (CRIT) . 
The UIT is implemented in accordance with the. above equation wherein 
threshold computed values are employed and wherein an ith component of a 
state vector x L at time t+1 is given by 

M 

x i (t+ 1) = £ < a m (t) +J i , 

in-1 


where the terminal attractor is given by 

I ± = -f;«"(t)[g(x J (t))-ug 1/3 5-( t), 

ill-1 


b m {t) = exp 


N 

F 

V i- 1 


/nl2 




N 


a m (t) = vjgr(xj(t ) ) , 


g(Xj(t)) = 1/ (l+exp(-a[Xj(t)-0 (t) ] )) , 

is a nonlinear sigmoidal function. The constant a in the sigmoidal 
function is an activation function which determines a slope of the nonlin- 
ear sigmoidal function and 0(t) is a threshold value given by 


M 


0(t) = £ a m (t) 


J7J-1 


— +6 m ( t) 


Thus, the UIT provides a thresholding means for implementation of a 
unipolar TABAM. For the CRIT, the model is given by 

x^t+1) =£ a m (t)6 m (t)[(l+6“(t))Uj+g(x J (t))6 m (t)]. 

/n-1 


This equation indicates that the crosstalk between the nonorthogonal stored 
vectors is exponentially weighed and reduced by the exponential term 6 (t) . 
The property of the basin of terminal attractors is controlled by the value 
of fi in 6 m ( t ) . Using this CRIT model, the retrieved state vector is highly 
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rx 


likely to be placed in a correct basin. The threshold can be set in a 
manner similar to that used for the UIT as follows. 


M 


0 ( t) = £ a m (C) d m (C) 
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FIG. 1 is a plot of tf = u k to show phase flow in the neighborhood of 
reqular and terminal attractors. FIGs. 2a and 2b are respective plots of 
relaxation time, t if versus initial value of u - u ± for regular and 
terminal attractors. FIG. 3 is a diagram for the architecture of a 
terminal-attractor based neural network using a photodetector array PDA, 
a sigmoidal function g(Xj) and a connectivity mask T i;j . FIGs. 4(a) and 
4(b) illustrate diagranunatically the architecture for two optoelectronic 
neural networks employing electronically addressable and programmable 
spatial light modulators ( SLM) panels for TABAM implementation with one- 
dimensional input. FIG. 5 is a schematic diagram of an optoelectronic 
implementation of a TABAM with two-dimensional input neuron-state vectors. 
FIG. 6 illustrates diagranunatically the inner-product operation using two 
transmissive liquid-crystal SLMs . FIG. 7 illustrates a dynamic logistic 
(sigmoidal) function with a sharp transition at x l = 0 (adaptive thresh- 
old). FIG. 8 illustrates in a graph how the adaptive threshold, 0, can be 
set to achieve the maximum noise immunity in the system, and. FIG. 9 
illustrates an experimental set-up used to demonstrate the feasibility of 
the optical XOR operation in the system of FIG. 10. 

The novelty of the invention resides in the provision of the terminal 
attractor I • . That is basic to the UIT and the CRIT because the terminal 
attractor, ^Iv used in each iteration. Consequently, basic to this 
invention is a TABAM which, unlike that of Zak, supra, can be implemented 
optically to provide a neural dynamic system having finite relaxation 
times, no spurious states, and virtually infinite stability for use in 
real-time, high-density associative memory applications. 
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UNIPOLAR TERMINAL-ATTRACTOR BASED NEURAL 
ASSOCIATIVE MEMORY WITH ADAPTIVE THRESHOLD 

OPTCTN OF INVENTION 

The invention described herein was made in the 
performance of work under a NASA contract, and is 
subject to the provisions of Public Law 96-517 (35 USC 
5 202) in which the contractor has elected not to retain 

title . 

TKCHNICAL FIELD 

The invention relates to a neural associative 
memory system based on nonlinear, dynamical terminal 
10 attractors with adaptive threshold values for dynamic 
iteration of unipolar, binary neuron states. 


BACKGROUND ART 

One of the major applications of neural networks 
is in the area of associative memory. The avalanche 
15 of intensive research interests in neural networks was 
initiated by the work of J.J. Hopfield, "Neural Net- 
works and Physical Systems with Emergent Collective 
Computational Abilities," Proc. Nat. Acad. Sci, USA, 
Vol . 79 p. 2254-2258 (1982), U.S. Patent No. 4,660,166 
20 in which the associative memory is modeled with a 
neural synaptic interconnection matrix and encompasses 
an interesting computation scheme using recursive, 
nonlinear thresholding. Further investigation report- 
ed by R.J. McEliece, E.C. Posner, E.R. Rodemich, S.S. 
25 Venkatesch, "The Capacity of the Hopfield Associative 
Memory," IEEE Transactions on Information Theory, Vol. 
T-33, pp. 461-482 (1987); and B.L. Montgomery and 

B.V.K. Vajaya Kumar, "Evaluation of the use of Hop- 
field Neural Network Model as a Nearest Neighbor 
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Algorithm," Appl. Opt. Vol. 25, pp. 3759-3766 (1986) 
reveals that the storage capacity of the Hopfield 
Model is quite limited due to the number of spurious 
states and oscillations. 

5 In order to alleviate the spurious states prob- 

lems in the Hopfield model, the concept of terminal 
attractors was introduced by M. Zak, "Terminal Attrac- 
tors for Addressable Memory in Neural Networks," Phys. 
Lett. Vol. A-133 , pp. 18-22 (1988). However, the 

10 theory of the terminal-attractor based associative 
neural network model proposed by Zak determines that 
a new synapse matrix totally different from the 
Hopfield matrix is needed. This new matrix, which is 
very complex and time-consuming to compute, was proven 
15 to eliminate spurious states, increase the speed of 
convergence and control the basin of attraction. Zak 
(1988), supra, and M. Zak, "Terminal Attractors in 
Neural Networks," Neural Networks, Vol. 2, pp. 259-274 
(1989) 

20 Zak's derivation shows that the Hopfield matrix 

only works if all the stored states in the network are 

orthogonal. However, since the synapses have changed 
from those determined by Hebb's law, Zak's model is 
different from the Hopfield model, except for the 

25 dynamical iteration of the recall process. The 

improvement of the storage capacity of the Hopfield 
model by the terminal attractor cannot be determined 
based on Zak's model. 

More recently, for the purpose of comparing the 
30 Hopfield model, both including and excluding a termi- 
nal attractor, a terminal-attractor based associative 
memory (TABAM) model has been proposed which incorpo- 
rates binary neurons into the synaptic matrix deter— 
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mined by Hebb's law in the same way as the Hopfield 
model. That work has been disclosed in a paper by 
H.K. Liu, J. Barhen and N.H. Farhat, "Optical Imple- 
mentation of Terminal Attractor Based Associative 
5 Memory," Appl. Opt., Vol. 31 , pp. 4631-4644, August 
10, 1992, which by this reference is incorporated 

herein and made a part hereof. Among the several 
techniques proposed for optical implementation of the 
terminal attractor, the most important includes the 
10 application of the inner-product approach first 

proposed by S.Y. Kung and H.K. Liu, "An Optical Inner- 
Product Array Processor for Associative Retrieval," 
Proc. SPIE , Vol. 613 , pp. 214-219 (1986) and later 

fully implemented by H.K. Liu, U.S. Patent Application 
15 Serial No. 07/880,210 titled "Optical Inner-Product 
Neural Associative Memory," which by this reference is 
incorporated herein and made a part hereof . Also 
included among the several techniques proposed for 
optical implementation of the terminal attractor is 
20 the exclusive-or (XOR) operation of the liquid crystal 
television spatial light modulator (LCTV SLM) de- 
scribed by H.K. Liu and T.H. Chao, "Liquid Crystal 
Television Spatial Light Modulators," Appl. Opt., Vol. 
28 , pp. 4772-4780 (1989) . 

25 The complexity of the optical implementation of 

a TABAM is discussed in the 1992 paper by Liu, Barhen 
and Farhat. In general, optical implementation of 
subtraction increases the complexity of a TABAM 
system. Consequently, a unipolar neuron model, which 
30 has only 1 and 0 as binary states, instead of 1, 0 and 
—1 would be more suitable for implementation of a 
TABAM, but that would require optical implementation 
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without subtraction or negative numbers normally 
encountered in a TABAM. 

STATEMENT OF THE INVENTION 

An object of this invention is to provide a TABAM 
5 system which, unlike the complex terminal attractor 
system of Zak, supra, is not defined by a continuous 
differential equation and therefore can be readily 
implemented optically. 

A further object is to realize such a TABAM by an 
10 optical implementation of an inner-product neural 
associative memory. 

Yet another object of this invention is to 
provide a unipolar inner-product TABAM (UIT) and a 
crosstalk reduced inner-product TABAM (CRIT) . The UIT 
15 provides a thresholding means for implementation of a 
unipolar TABAM. The CRIT provides a means for trans- 
forming the input state vector into the correct basin 
of the stored vector; then it uses the terminal at- 
tractor to accelerate the convergence. In both the 
20 UIT and CRIT, a dynamical logistic function is provid- 
ed for adaptive thresholding the output of each 
dynamical iteration of the recall process. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a plot of 0. = u k to show phase flow in 
25 the neighborhood of regular and terminal attractors. 

FIGs. 2a and 2b are respective plots of relax- 
ation time, t if versus initial value of u = u i for 
regular and terminal attractors. 

FIG. 3 is a diagram for the architecture of a 
terminal-attractor based neural network using a 


30 
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photodetector array PDA, a sigmoidal function g(Xi) 
and a connectivity mask ^ j . 

FIGs. 4(a) and 4(b) illustrate diagrammatically 
the architecture for two optoelectronic neural net- 
5 works employing electronically addressable and pro- 
grammable spatial light modulators (SLM) panels for 
TAFftM implementation with one-dimensional input. 

FIG. 5 is a schematic diagram of an optoelec- 
tronic implementation of a TABAM with two-dimensional 
10 input neuron-state vectors. 

FIG. 6 illustrates diagrammatically the inner- 
product operation using two transmissive liquid- 
crystal SLMs. 

FIG. 7 illustrates a dynamic logistic (sigmoidal) 
15 function with a sharp transition at x I = 6 (adaptive 

threshold) . 

FIG. 8 illustrates in a graph how the adaptive 
threshold, 9, can be set to achieve the maximum noise 
immunity in the system of FIG. 10. 

20 FIG. 9 illustrates an experimental set-up used to 

demonstrate the feasibility of the optical XOR opera- 
tion in the system of FIG. 10. 

FIG. 10 illustrates an optical inner-product 
neural network system with thresholding in accordance 
25 with the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

In the neural network associative memory model 
introduced by Hopfield, supra, the number of vectors 
that can be stored are limited. The capacity estima- 
30 tor by McEliece, supra, was based on the assumption 
that m of the randomly chosen n-tuple binary bipolar 
vectors are used for storage. When the network is 
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probed by one of the possible n-tuple vectors which 
are less than a Hamming distance n/2 away from the 
fundamental memory, the maximum asymptotic value of m 
can be no more than n/(4 log n) as n approaches 
5 infinity. 

The limitation of the capacity of the Hopfield 
model is partially due to the existence of spurious 
states. The spurious states arise in several forms. 
Most commonly seen spurious states are those stable 
10 states that are not originally stored. In terms of 
phase space terminology, these are false attractors 
trapped in the local minima in the energy landscape. 
Another problem is that there are states that are 
supposed to converge to certain stored states, but 
15 instead converge to other stored states that are more 
distant. Further, Montgomery and Kumar (1986), supra, 
pointed out the existence of oscillating states that 
also affect storage capacity. However, by letting the 
diagonal terms be naturally determined in the inter- 
20 connection matrix instead of setting them to zero, the 
oscillating states mostly disappear. In addition, the 
model is found to be convenient for optical implemen- 
tation. J.S. Bayley and M.A. Fiddy, "On the use of 
the Hopfield model for optical pattern recognition," 
25 Opt. Commun., Vol. 64, pp. 105-110, (1987); G.R. 

Gindi, A.F. Gmitro and K. Parthasarathy , "Hopfield 
model associative memory with nonzero-diagonal terms 
in memory matrix," Appl. Opt., Vol. 27, pp. 129-134 
(1988); and H.K. Liu, T.H. Chao, J. Barhen and G. 
30 Bittner, "Inner-product optical neural processing and 
supervised learning," First Annual Conference of the 
International Neural Network Society (INNS), Boston, 
Mass., September 6-10, 1988. 
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To increase the storage capacity of the neural 
network, one must reduce or eliminate the spurious 
states. A new type of attractor called a terminal 
attractor, which represents singular solutions of a 
5 neural dynamic system, was introduced by Zak (1988, 
1989), supra , for the elimination of spurious states 
in the associative memory. These terminal attractors 
are characterized by having finite relaxation times, 
no spurious states, and infinite stability. They 
10 provide a means for real-time high-density associative 
memory applications and potential solutions for 
learning and global optimization problems. 

The Zak terminal-attractor model assumed continu- 
ous variable representation of neural states in the 
15 neural dynamic equations. Also, sigmoidal thresh- 
olding functions were assumed. These assumptions 
present difficulties for optical implementations. To 
implement the terminal-attractor model, modifications 
to the basic Zak model are necessary. 

' 20 The purpose of the following is to describe 

briefly the basic idea of terminal attractors, to 
modify these attractors for binary neural state 
representations for associative memory, and then to 
disclose relevant optical implementations . 

25 I. Theoretical Discussions 

In the following theoretical discussions, the 
basic characteristics of the terminal attractor is 
first briefly presented, and then it is shown how to 
apply the attractor to nonlinear neural dynamic 
systems . 


30 
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A. Terminal Attractors vis-A-vis Regular Attractors' 
The concept of terminal attractors described by 
Zak (1988, 1989), supra, used a differential equation, 

d = u k . (1) 


It follows that 


du 

du. 


= -ku*' 1 , 


( 2 ) 


5 which is plotted in FIG. 1 with k as a parameter. 

It k > 1, du/du=0 as u-*0, and u=0 is called a 
regular attractor. In contrast, if k * 1/3, 1/5,... 
1/ ( 2 / 1 + 1 ) , then 

00 as u-0 . 

du 


Therefore u=0 is called a terminal attractor, which is 
10 used to describe rapid convergence to an attractor 
point. 

The relaxation time for a solution with the 
initial condition u 0 is 

t 0 = r° ( du/u k ) 

J u 0 


U 


1 -k 




(1 -k) 
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l-Jt 


U ° - - lira 
(1-ic) u-0 


u 


1-Jt 


1 -k 


(3) 


From Equation (3), it is seen that t 0 -»°° as u-0 for 
k>l and 


for Jkr= 1 / 3 , 1/5,..., l/2n+l) or simply for k<l in 
5 general. 

In FIGs. 2a and 2b, the relaxation time is 
plotted versus the initial values of u=u^ for both the 
regular and the terminal attractors, respectively. It 
is clear that the relaxation time approaches infinity 
10 for the regular attractors, meaning that u never 
intersects the solution u=0. In contract, for the 
terminal attractors, u/t=0J=u i decays to u=0 at 
t=t^,u(t=0)=u 2 decays to 0 at t=t 2 , and so on, all in 
finite times. This decay distinguishes the terminal 
15 attractors from the regular attractors. However, the 
Lipschitz condition for Equation (1) is violated for 
k<l, as is seen from Equation (3). 


B. Terminal Attractors for Associative Memory 

As just noted above, one of the fundamental 
20 problems of the Hopfield associative memory model is 
the existence of spurious states or the trapping of 
neural states in local energy minima of the phase 
space. For this reason, statistical mechanical 
models, such as simulated annealing, have been pro- 
25 posed to permit uphill jumps in the phase space so 
that the state of the network can be pumped out of the 
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local minima in order that it eventually reach the 
global minimum. In spite of the fact that this is a 
slow procedure, it is useful for applications in 
solving optimization problems. Therefore, the follow- 
5 ing discussion is confined to the use of terminal 
attractors to solving associative memory neural 
networks . 

Let X=[x m ,m=l , . . .AT] denote a set of M linearly 
independent vectors in J? N ; then assume that the neural 
10 network for the memorization of X is governed by the 
following nonlinear dynamic equation: 

+ a i x i = E - E 

j- 1 /7T-1 

x exp [-P” (Xj - X?) 2 ] , (4) 

where and | are positive constants and 

k=(2n+l )~ 1 where nil. For simplicity and without 
losing generality, assume that n=l and k= 1/3 for the 
15 terminal attractor. The term is the ith row and 
jth column element of the weight (synapse) matrix, and 
is a threshold function. A sigmoidal function is 
usually chosen for to satisfy the differentiability 
requirements for the continuous variables. 

20 According to the work of Zak (1989), supra, the 

addition of the terminal attractor term in Equation 
(4) has to be based on the following conditions in 
nonlinear neural dynamics: (a) the Lipschitz condi- 

tion at a fixed point is violated and (b) the Hop- 
25 field-type memory synaptic interconnection matrix is 
generally no longer valid unless the stored states are 
all mutually orthogonal. However, in the present 


F93173 


11 


invention, the Hopfield matrix is kept the same as 
that shown in Equation (4) for the following reasons. 
The exact reason for evoking the terminal attractor is 
to see whether the new term can improve the storage 
and retrieval capability of the neural system as 
compared with the system without the new term. If the 
memory matrix term in the model is changed, a highly 
sophisticated computation of the matrix itself must be 
made which, according to the theory derived by Zak, is 
just as complicated as the computation of the whole 
model. Hence, not only can optics hardly be made 
feasible for the implementation of the new model 
because nonlinear differential equations are involved, 
but also the impact of the terminal-attractor term on 
the Hopfield model cannot be determined because the 
new system is no longer related to a Hopfield neural 
network. 

For optical implementation, there is already a 
large amount of research available that is based on 
the Hopfield model. The simple addition of the 
terminal-attractor term, as shown in Equation (4) , 
represents a modification of the Hopfield model and 
the effort can therefore be focused on this part only. 
Nevertheless, the reason of expediency alone is 
insufficient because the basic theory is against such 
a simpler approach. To see whether the model of 
Equation (4) is justified, a computer simulation on 
the terminal-attractor based associative memory 
(TABAM) model was made with unipolar binary-neuron 
states and an adaptive threshold as described below. 
The computer-simulation results show that at least 
with a small number of neurons and with an almost 
equal number of stored states, perfect convergence to 
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the correct stored states is achieved based on an 
exhaustive search with all possible input states and 
all combinations of possible stored states. In the 
computer simulation, the value of n=l (and therefore 
£=1/3) was used for the terminal-attractor term. 
Because of the choice of a larger n, for different k 
values, the computation would be more complicated and 
the optical implementation more difficult. However, 
a larger number of neurons could be selected with 
mainframe computers for the simulation. The present 
simulation with n=l offers sufficient reason to 
proceed with an optical implementation of the TABAM. 

C. Binary-Data Representation for the TABAM 
1. Basic Assumptions 

In the hardware implementation of neural networks 
for associative memory, the input pattern and output 
pattern are usually represented in binary form. In 
the Hopfield model, bipolar binary representation is 
usually assumed, and so the stored vectors are given 
in binary form. However, in Equation (4) it has been 
assumed that the stored vectors have continuous values 
and that gj(Xj) also has continuous values, so modifi- 
cations are required before this equation can be used 
for associative memory implementation. A modified 
model is given below. 

Assume that a set of binary W-tuple vectors 
V={ u m ,m=l , 2 , . . . ,M} are stored as terminal attractors 
in the neural net. The neural dynamics are described 
by 

*i * *1*1 = E T if g{ Xj ) - £ a?[ g(x i ) - <] ^ 

j^l m-1 
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x exp{-Pi[g(x i ) - u j] } . ( 5 ) 

Also assume the sigmoidal function: 

g(Xj) = tanh axj; ( 6 ) 

lima^grpc-j) is then a binary component. 

2. Stability 

The proof of stability of the binary-data-repre- 
sented TABAM requires a linear approximation of 
Equation (5). The linearization is accomplished by a 
first-order approximation of a Taylor-series expan- 
sion : 

w 

+ a 1 x i + T T Lj lim g{x j ) 

j*i a — 

+ lim — — (x 1 -x i rn ) -a™ 
a-* cosh 2 ax i 


x exp j-Pitlim g-(x i )-Uj ] 2 


F93173 


14 


x 


lim 

a-«* 


a 

cosh^Xj 



x 



x exp [-P7 (Ui-Ui / ) 2 ] '(xj-xf) . (?) 


x 


(Xj-Xf) 




“ 2/3 


X lim 2pf 

a ~“ cosh 2 ax" 


Note that at x^x^ and at g(x±)-*v™ as a-»<», 

Xj = ( 00 )^+ finite terms 

Hence the system is infinitely stable at x i =x i m . 

D. Assumptions for the Optical Implementation of the 
5 Terminal-Attractor Based Associative Memory 

The above proof of stability for the binary- 
number-represented terminal-attractor based neural 
dynamic system is essential before hardware implemen- 
tation is considered. Further assumptions will be 
10 required for the optical implementation of the TABAM. 

These assumptions are related to the determina- 
tion of the values of T ljt a m , P™, and the threshold 
used in the function q() in Equations (5) and (6). 
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First we discuss T ^j. The simplest assumption is 
that j may follow Hebb's rule used in the Hopfield 
model : 


H 


Tij = £ < u j- 


/n-l 


( 8 ) 


When this term was used in the Hopfield model, spuri- 
5 ous states were found. For the sake of comparing the 
effects of the reduction of spurious states in the 
TABAM on the same basis, the same assumption should be 
kept. Other learning rules may be used with the TABAM 
for Tj^j, such as the one suggested by F.J. Pineda, 
10 "Generalization of back-propagation to recurrent 
neural networks," Phys. Rev. Lett., Vol. 59, pp. 2229- 
2232 (1987). However these T^'s will require exten- 
sive computations by digital computer as well as a 
decision on whether the spurious states exist in these 
15 models when the terminal attractors are not invoked. 
Therefore, the following discussion leaves the options 
of the selections of T^j open for any future implemen- 
tation schemes. 

The purpose of P™ and the exponential term in the 
20 terminal attractor is to provide a Gaussian distribu- 
tion peaked at gix^ = u". When g(x x ) moves away from 
u i , the exponential function decays to zero. This 
term maximizes the influence of the terminal attractor 
for g^Xj) that approaches u™. However, it also adds 

25 complication and difficulty to optical implementa- 
tions . 


F93173 


16 


If Pi = 0 , the exponential term becomes 1. This 

is equivalent to stating that g(Xj) = Vf for all m. 
This is obviously impossible unless there is only one 
stored state, i.e., M — 1. Therefore, Pj>0. 

5 One way of determining the parameter a™ is given 

as follows. In the Hopfield model, the activation- 
potential component x L in the iteration is given by: 

x'i - E T ij?( x j) 

J-l 



M 




(9) 


One may rearrange the terms and obtain an inner- 
product formalism of this model by letting 

m N 

x'i - E u " E u i S r(x J ) 

171-1 J-l 





177-1 


( 10 ) 


10 where 
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<X m 



g(xj) , 


which is the inner product between one of the stored 
vectors and the computed vector. The parameter is a 
measure of the degree of similarity between the 
computed vector and one of the stored vectors. 

5 To emphasize the impact of the terminal attractor 

represented by a stored vector that is most similar to 
the computed vector during iteration, it is suggested 

that the parameter a” be chosen as 

a" = a'", i = 1, . . . ,N. (11) 

With this choice, the stability of the dynamic 
10 system is not affected because the term leading to 

infinity at gixj = u™ is not changed. With this 

selection of parameters and without loss of generality 
by letting ^ * 1 at steady state = 0) , Equation 

(5) may then be written as 

xl = T T^gUj) -£ a m [g{ Xj ) - U JJ ^ 

m-l 


x expt-p^srU^-u?] 2 } 

= T ij9ix j ) 


( 12 ) 
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Jj = Ea”[^(x i )-<] 1 / 3 exp{-p”[g^(x i ) -u"] 2 } . 

m-1 

Equation (12) is generic in the sense that is not 
specified. 

In the following discussion, further approxima- 
tions are made and a simpler model that is more viable 
5 for optical implementations is derived. Assume that 
the Hebbian rule as shown in Equation (8) is used for 
IV^j. Then based on Equations (10) and (11), Equation 
(12) may be written as 

M M 

x[ = £ ccV?-£ a. m [g { Xi ) -<] ^ 

m-1 m-i 

X exp{-p?[g(x i )-u?] 2 } . (13) 

Pi will now be considered. In the same spirit of 

10 determining a™, it is assumed that the exponential 

term in Equation (13) may be set in a collective 
manner : 

exp{-p™[g(x i ) -Uj] 2 }-exp{-p m Y) [g( Xi ) -<] 2 } , 


P™ = P" 1 = a constant 

In addition, if it is assumed that unipolar binary 
numbers are used for the neuron-state vectors, the 
factor of 1/3 in the exponential term of Equation (13) 


15 
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may be dropped without changing the value of the 
equation. 

Equation (13) may then be written as 


M 


a m u™+]T a m exp 


/n«l 


M 

C 

/n«l 


-P m £ [s r u i )-<]4x[u^u 1 )] . (i4) 


j-i 


In terms of vector representation. Equation (14) may 
be written as 

M 

x' = £ [ V m, g(X) ] 

/n»l 


x (^ n, + {exp[-P m |u fl, -^(X) | 2 ] } [V m -g{X )] ) , (15) 


where X' is the updated neuron activation-potential 
vector and g(X) is a thresholded vector where the 
function g is applied to every term of X in taking the 
threshold. 

Equation (15) involves the inner product of V™ 
and g(X) and the subtraction between V 1 " and g(X) , but 
it is only valid for unipolar binary representation of 
the vectors. This will be noted again in the discus- 
sion below of a unipolar TABAM with adaptive threshold 
and perfect convergence. 

The convergence of the iteration not only depends 
on the data representation, the number and the content 
of the stored vectors, the initial input, the synap- 
ses, and the parameters a m and j8 m , but also critically 
depends on the thresholding value in the function g. 
In binary-state representations, it is most convenient 



F93173 


20 


to use the sigmoidal function as the thresholding 
function, but it can be shown that convergence depends 
on the threshold. Unfortunately, it is quite diffi- 
cult to determine the threshold value based on rigor- 
ous analytical results. Therefore, with the assis- 
tance of computer simulation, the empirical method 
will be used for the selection of the threshold. 

II. Optical Implementation of the Terminal-Attractor- 
Based Associative Memory [TABAMJ 

Two types of optical architectures for the 
implementation of the TABAM will now be discussed. 
One type is based on the multiplication of a one- 
dimensional input vector with a two-dimensional 
matrix; the other uses two-dimensional input vectors. 
The architectures described are based on state-of-the- 
art optics and electronics hardware described by, for 
example, N.H. Farhat, "Optoelectronic analogs of self- 
programming neural nets," Appl. Opt., Vol. 26, pp. 
5093-5103 (1987)] 

A. Optical TABAM with One-Dimensional Input Neuron- 
State Vectors 

If the input vectors are in one-dimensional form, 
a reasonable approach to hardware verification of the 
TABAM is shown in FIG. 3. Theoretically, the termi- 
nal-attractor term 1^ of Equation (12) may be repre- 
sented by the light-emitting diode array LEA2, and the 
one-dimensional input vector is applied by the 
light-emitting diode array LEA1 represented by a row 
of dots, one dot for each diode. The indices i and j 
run from 1 to N. These LEA's may be replaced by 
acousto-optic cells with coherent illumination. The 
output is precisely as given by Equation (12). 
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Two embodiments of an alternative implementation 
of the TABAM are shown in FIGs. 4a and 4b. In FIG. 
4a, two light (charge) integrating photodetectors PD + 
and PD" are connected to a differential amplifier DA 
to provide the output X ' Both spatial light modula- 
tors SLM1 and SLM2 are liquid crystal TV (LCTV) panels 
electronically addressable and programmable with SLM2 
displaying ternary connectivity mask T ± j . It should be 
noted that SLMl is shown with a first vector stored 
on the top two rows and subsequent vectors stored on 
succeeding pairs of rows, and that of the top row in 
each pair of rows is focused onto the photodetector 
PD + and the bottom row onto the photodetector PD". The 
reasons for this will become apparent from the de- 
scription of operation below. 

In FIG. 4b implementation is the same except a 
self-scanning charge-coupled device array CCD is used 

with odd-even photosite outputs for the output x\ . 

The advantage of this CCD for the output Xj will be 

evident from the description of operation below. 

Thus, in FIGs. 4a and 4b, two electronically ad- 
dressed spatial light modulators (SLMs) designated 
SLMl and SLM2 are used to represent the state vector 
v(j f j = 1,2. ..W; and the connectivity matrix T L ^,{i f j 
= 1,2...N ) respectively, with N being the number of 
neurons in the network. One column on the SLM2 is 
used for the input of I if which is calculated by a 
computer. Assume that both SLMs can be refreshed at 
a video rate from at least one computer controller CC 
with video memory. The two SLMs are identical and are 
deployed in tandem (proximity coupled) by aligning 
them so that pixels are in perfect registration. By 
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using collimated spatially incoherent light for the 
illumination, multiplication of the transmittance 
patterns stored in the two SLMs is achieved. A 
coherent fiber-optic faceplate can also be used to 
proximity couple the two SLMs to minimize crosstalk 
between pixels if this process proves to be problemat- 
ic . 

The state-of-the-art commercial SLMs, such as the 
liquid-crystal TV (LCTV) spatial light modulator, are 
available with approximately 200 x 200 pixels and an 
achievable contrast of at least 30:1 on a rectangular 
grid format. This availability permits the construc- 
tion of neural network modules with an order of 100 
neurons in these architectures. Larger neural net- 
works can be formed by clustering several such mod- 
ules. Clustering is expected to lead to some sacri- 
fice in overall speed, but this may be an acceptable 
price to pay for the advantage of being able to form 
flexible neural networks of a few hundred to a thou- 
sand neurons with available SLM technology. 

In addition to the LCTV SLMs mentioned above, 
other SLMs are also commercially available. A promis- 
ing candidate for the electronically addressable SLM 
in these architectures is the Semetex 256 x 256 pixel 
magneto— optic SLM (MOSLM) . A minimum contrast ratio 
of 10000:1 has been reported for the MOSLM. Another 
device that has recently become available is the 
ferroelectric liquid-crystal SLM, which has 128 x 128 
pixels, 100-Hz driving speed, and a contrast ratio 
greater than 100:1. However, all of these devices 
except the LCTV SLM are basically binary or trinary in 
nature. If multiple values of the interconnection- 
matrix component are to be represented, quantization 
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would be needed. In that case, more pixels would be 
required to represent one analog value, and the total 
number of neurons that can be processed would be 
reduced. The main drawback of the LCTV SLM is its 
limited dynamic range and contrast ratio. However, 
the combination of thin-film transistor technology 
with special waveforms for driving pixels in the LCTV 
SLMs has resulted in improved contrast and dynamic 
range levels that are more suitable for realizing the 
•^ij m atrix in these architectures. 

In the hardware operation, two LCTV SLMs (or 
MOSLMs) or ferroelectric liquid-crystal SLMs can be 
used in the first embodiment of FIG. 4a to realize a 
neural network of approximately 60 neurons. The state 
vectors are displayed on SLM1 in such a way that each 
element of the unipolar binary-state vector i/j, j » 
1,2 . . .N where N = 60 is assigned to four adjacent 
pixels in two adjacent rows of SLMl. The vector i/j is 
displayed by using the two top rows to obtain the 
activation potential of the first neuron in the 
network. If the LCTV is used, its large composite 
pixel size has the advantage of making alignment easy, 
thus minimizing diffraction by pixels. Therefore, 
crosstalk is reduced between pixels in the event that 
proximity coupling of SLMl to SLM2 is chosen in the 
implementation. (Otherwise, an imaging lens can be 
used for the multiplication.) 

SLM2 is used to display the connectivity weights, 
in the geometry shown. It is assumed that the dynamic 
range of the SLM is sufficient to accommodate the 
spread in weight values, which for the Hopf ield-type 
of storage ranges between -AT and +M where M is the 
number of entities. The bipolar values of the connec- 
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tivity matrix are encoded by assigning positive values 
to one row of SLM2 and negative values to the adjacent 
row. The terminal-attractor term is first computed 
in accordance with Equation (12) and then applied to 
5 the far-right column of SLM2. Light emerging from 
these rows is focused with the aid of appropriate 
optics (not shown) onto photodetectors PD + and PD” in 
such a way that PD + collects light emerging from the 
positive row and PD" collects light emerging from the 
10 negative row of T L j. The photodetectors PD + and PD” 
are followed by the differential amplifier DA compris- 
ing two charge-integrating amplifiers whose outputs 
are subtracted before the difference is sampled and 
stored by the computer controller CC through its 
15 analog-to-digital converter port. The outputs can be 
subtracted at the instant the writing of the first two 
rows of SLM1 and SLM2 is completed or at the end of 
writing one complete frame in SLM1 and SLM2 . This 
subtraction ensures that the quantity sampled by the 
20 computer is the activation potential of the first 
neuron: 

= E T ijVj +I i' i-i. ( 16 > 

To sample and store the activation potential of all 
other neurons in the net, the position of the two rows 
of SLMl on which Uj is displayed is lowered in steps 
25 of two rows during each iteration. For each frame or 
display of i/j, is also displayed on SLM2 and the 

activation potential is measured and stored as previ- 
ously described. In this fashion, and as the 
display is moved from the top two rows to the bottom 
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two rows of SLM1, the activation potentials for all N 
neurons are sequentially generated, sampled, and 
stored by the computer-controller. The time needed to 
accumulate the activation-potential vector for these 
N neurons equals N times the frame time of the SLMs. 
For the LCTV SLM, the time is N x 30 ms or 1.8 s. 
Once the activation-potential vector x' is measured, 
the computer-controller CC computes the updated state 
vector for the neural network: 

u| g+1) =g{x’i {q) } i=l,2, . . .N, g=0 ,1,2,3..., (17) 


where designates the sigmoidal function of the 

bracketed quantity . The updated state vector is used 
next with the same procedure for generating the 
activation-potential vector described above to obtain 
a new state vector and so on. The above process is 
repeated until the state vector converged to a stable 
state. When this happens, the state vector of the 
neural network in phase space or configuration space 
is said to have converged to an attractor of the 
neural network. 

Before describing the second embodiment of FIG. 
4b, note that the charge-integrating amplifiers in the 
first implementation may not be required because of 
the long pixel-relaxation time of the SLM, which is 
typically of an order of 10 ms but can be as high as 
100 ms in some devices. Thus, in this case photode- 
tectors PD + and PD" measure the activation potential 
directly without the need of a time-integration 
device, thereby simplifying the implementation pro- 
Addition simplification would be to present 


cess . 


F93173 


26 


outputs of PD + and PD" directly to the computer- 
controller where each is separately converted from 
analog to digital form before the computer-controller 
carries out the subtraction. 

To speed up the operation and convergence time in 
the first embodiment, it is possible to modify the 
architecture as shown in FIG. 4b. In this embodiment, 
a self-scanning CCD is used as a PD array to measure 
the activation-potential vector in one video frame 
time instead of the N frame times needed for the first 
scheme. An N time-speed enhancement is therefore 
achieved, which assumes that the CCD PD array can be 
read out in one video frame time or less. CCD PD 
arrays are available with bandwidths of 20 MHz or 
more. This availability means that a device with 120 
photosites each receiving light focused from the 120 
rows of the in SLM2 of the f irst embodiment can be 
read in 6 ps or less, which is considerably shorter 
than the 30 ms frame time of the utilized SLMs. The 
state-update time of this scheme would therefore be 
primarily limited by the frame time of the SLMs used, 
or 30 Hz for the LCTV SLM. Note that CCD photosensor 
arrays with a separate odd— even photosite readout, 
such as that of the Recticon CCD arrays, are commer- 
cially available. This readout feature would facili- 
tate subtracting the outputs of positive and negative 
photosite pairs to obtain the activation potential of 
the neuron. 

B. Optical TABAM with Two-Dimensional Input Neuron- 
State Vectors 

On one hand, architectures with one-dimensional 
input neuron-state vectors are useful for the process- 
ing of signals that are originally presented in one- 
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dimensional form, such as radar, sonar, or seismic 
wave data. Optical images, on the other hand, are 
normally received in two-dimensional form. Although 
two-dimensional images can be converted by raster 
scanning to one-dimensional signals before processing, 
a two-dimensional reconstruction is then required to 
restore the image after processing. The processes 
must have additional processing time and hardware. 
For this reason, two architectures of the TABAM for 
processing two-dimensional input neuron-state vectors 
are proposed as follows. 

N 4 to N 2 Interconnection Architecture 
When the input neuron-state vectors are repre- 
sented by a two-dimensional form of N 2 pixels, the 
interconnection matrix requires N 4 pixels. In this 
case. Equation (12) is modified as 

X ij ~ 53 52 T ijkl&( X kl) 

k -1 1-1 


M 


- £«-[^U w )-u ^] 1/3 


m-l 


- exptp^teU^-u^] 2 }, (18) 


where g(x^) denotes a threshold ith row and jth column 
component of a neuron-state vector arranged in a two- 
dimensional matrix, and represents a component of 
the AT 4 interconnection matrix. 
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Recently a N ^ to N * architecture was discovered 
and experimentally demonstrated by T. Lu, S. W, X. Xu 
and F.T.S. Yu, "Optical implementation of programmable 
neural networks," Optical Pattern Recognition , H. Liu, 
ed., Proc. Soc . Photo-Opt. Instrum. Eng., Vol. 1053, 
pp. 30-39 (1989). This architecture can be modified 
to implement the TABAM with a two-dimensional input. 
As shown diagrammatically in FIO. 5, the weight matrix 
is presented in an N x N array with elements 

■Til kl • ^12 ^21 kl ’ ^22 kl' ' ' ' ^mi kl • each of which also 
has N x N pixels. The input neuron-state vector is 
represented by an SLM with N x N pixels. The inter- 
connection between the weight matrices and the input 
state is accomplished through the lenslet array L u , 
Ir 12 , and so on, where Ljj connects j^to the input, 
and so on. 

The terminal-attractor term is computed by a 
digital computer and added to the output term detected 
by the N x N PD array. The sum is thresholded before 
it is applied to the input SLM. The system can be 
activated with any initial input vector. 

If a commercially available video monitor with 
1024 x 1024 pixels is used fore the weight matrix, a 
maximum of 32 x 32 neurons can be processed. The 
advantage of the system is the dynamic range and 
number of gray levels (256) available in the video 
monitor. 


Inner-Product Architecture 
An optical inner-product approach for neural 
processing was proposed earlier by S.Y Rung and H.K. 
Liu, supra, and fully implemented by H.K. Liu, Patent 
Application No. 07,880,210 as noted hereinbefore. 
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Before describing the inner-product approach, the 
outer-product approach will first be briefly described 
for contrast. Assume that an integral number M of N- 
tuple binary vectors V i (i=l,...M) are stored in a 
5 system where M«N. 

A computation model with iteration, thresholding 
and feedback for the associative retrieval of any one 
of the stored vectors by an arbitrary input vector was 
proposed by J.J. Hopfield (1982), supra. In the 
10 Hopfield model, the vectors are stored in an outer- 
product matrix: 

W = £ Vtf 

i-l 

where t denotes the transpose of the vector V i . 

In the recall process, an arbitrary input vector 
V is assumed, and the following steps are followed: 

15 Step (1): Vector-matrix multiplication: 

w v=z 

Step (2): Thresholding by an operator 6: 

V* = T 0 [Z] 

Step (3): Iteration: 

20 Replace the input vector V in Step (1) by V* and 

repeat until convergence is reached. The V* at that 
time is the correct output vector 

An optical implementation of the outer-product 
process was first demonstrated by Psaltis and Farhat, 
25 "Optical Information Based on an Associative Memory 
Model of Neural Nets with Thresholding and Feedback, " 
Optics. Lett., Vol. 10, page 98, et seq., (1985). 
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In contrast to the outer-product approach, the 
steps of the inner-product operation are: 

Step ( 1 ) : Compute a vector-vector inner product scalar 

a: 

5 otj = V/V 

where V - / is the ith stored vector, i=l,...,M, and V is 

the input vector. 

Step (2): Weighting: 

iVi 

10 Step (3): Vector summation: 

H 

i-l 

Step (4): Thresholding by an operator 9: 

Step (5): Iteration: accomplished by substituting the 
threshold Z of Step (3) for the input vector V and 
15 repeat. 

The thresholding and iteration operations are 
similar to those described in the outer-product 
process. Since the process involves basically the 
inner-product between the input vector and the M 
20 stored vectors, this process is called an inner- 
product model. An optical implementation of the 
inner-product model is discussed below as an example 
of this invention. 

By using Equation (15) it is possible to imple- 
25 ment the TABAM based on the inner-product approach as 
shown in FIG. 6, which will be described in detail 
following an experimental set-up shown in FIG. 10. In 
this case, the weight matrix does not have to be 
precalculated because no weight matrix is used. The 
lack of the explicit expression of the synapses makes 


30 



F93173 


31 


this approach different from the conventional concept 
of a neuron network. Nevertheless, both analysis and 
computer-simulation results show that the inner- 
product and outer-product of the Hopfield model are 
5 equivalent if in the latter model the weight matrix is 
not thresholded and the diagonal terms are not set to 
zero. 

One practical advantage of the inner-product 
approach is that with the same space -bandwidth product 
10 capacity of SLMs, the inner-product computation allows 
processing many more neurons than the outer-product 
computation of the Hopfield model where vector-matrix 
multiplication is performed. In contrast, some 
learning algorithms such as the error backpropagation 
15 for distributed weight correction cannot be applied 
because of the nonexistence of a weight matrix. 

Attention is now called to the implementation of 
the approximated model based on Equation (15). From 
an optics point of view, the exponentiation and 
20 subtraction shown in the equation are difficult to 
implement. In the optoelectronics approach, these two 
operations can best be handled by electronics. For 
easier implementation, Equation (15) should be treated 
further. It is rewritten as follows: 

M 

X '=£ [V m -gtf)] {l+exp[-p m |v m -gU) \ 2 ]}V m 

m-1 
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- £ v m -g(X) exp [-p m | v m -g r (J ! 0 | 2 ] gtf) . < 19 > 

JTT-1 

Equation (19) contains two types of operations, 
namely the inner-product operation and the exclusive- 

OR operation between V 1 " and g(X). In addition, for 
unipolar positive binary representation, it is shown 

5 that 

|V m - g(X) | ® \V m - G{X) | 2 

in each of their components. 

In an optical implementation based on the use of 
the LCTV SLM, i.e., based on the rotation of the 
10 polarization orientation of the input light by the 
twisted nematic effect of the liquid-crystal molecules 
in the SLM cell, by appropriately orienting the input 
and output polarizers, the inner product and the 
exclusive-OR between two inputs can be obtained. FIG. 
15 6 depicts the inner-product operation. The first and 

second inputs are written on SLMl and SLM2, respec- 
tively. For simplicity, only four cells are shown in 
this discussion. A numeral 1 shown on the cell means 
that the liquid-crystal molecules will not rotate the 
20 polarization orientation of the light passing through 
it. A numeral 0 shown on the cell means that the 
liquid-crystal molecules will rotate the polarization 
of the light passing through it by 90° (in an ideal 
case) . The arrows in space represent the orientation 
x 25 of the polarization of light. Under the conditions 
shown in the figure, when the polarizer P and analyzer 
A are oriented vertically and in parallel, the inner 
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product between the first input and the second input 
results as shown in Table 1. 

Table 1. Truth Table for an Inner Product Using the 
Polarization Effects in a Liquid-Crystal SLM 


Input 1 

Input 2 

Output 

1 

0 

0 

0 

1 

0 

1 

1 

1 

0 

0 

1 


Notice in Table 1 that 0 x 0 = 1 in the unipolar case 
is equivalent to -1 x -1 = 1 in the bipolar case. 
Because of this result, the operation is equivalent to 
the inner product between the two inputs, which is a 
measure of the degree of resemblance of the two 
inputs. In contrast, if the analyzer of FIG. 6 is 
oriented perpendicular to the polarizer, an exclusive- 
OR operation between the two inputs results. This 
relationship establishes 

|inputl-input2|=l- (inpufcl) -(input 2) , ( 20 ) 


where the multiplication dot denotes the inner prod- 
uct . 

Taking advantage of Equation (20), Equation (19) 
may be rewritten as 

[ v m -g(x )] (l+exp{-V m [N-V m -g(X )) }) V m 

jn-i 
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M 

- £ [V a 'g{X) ] exp{-p m [iV-V ,n *g-(^) ] }g(X) 

m-l 

M 

= ^ a m {l+exp [-p m (jlv-a m (Ar-a m ) ] } 

ro-1 

M 

- a^exp [-P'"(i\r-a'") ] g(X) , (21) 

7H-1 

where a m = V m ‘g{X) and N is the maximum value of 
V m -g(X) . Let 

a m exp [-p ,n (W-a m ) ] =y m , (22) 

then 

M 

X' = £ [ (a m +Y m ) V m -y m g(X) ] . (23) 

/n-1 

In considering the optical implementation of Equation 
5 (23) there are two challenging problems. The first is 

the realization of the exponential and subtraction 
parts in Equation (22). The second is the performance 
of the subtraction in Equation (23). Because the 
first operation is a scalar-value operation, it is 
10 feasible to simply let a digital computer such as a 
personal computer (PC) perform the subtraction. 

The subtraction in Equation (23) can be carried 
out much faster if it can be done optically in paral- 
lel. The condition under which that subtraction can 
15 be performed optically in parallel will now be ex- 
plored. By examining Equation (23) it is found that 
the result of the computation is normally greater than 
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1 and may be less than 0 in some cases, such as when 
a specific component of the stored vector is always 0 
but the corresponding component of g(X) is not 0. It 
will be recalled that by definition a m * y m . In 
5 addition, based on the assumption that only unipolar 
binary numbers are used to represent the neurons, the 
threshold of the analog value of X' must be taken. 
The transformed unipolar positive binary-neuron vector 
is then used for the succeeding iterating operations. 
10 If the threshold is set to be at least greater than 

M 

£ Y 1 ", Equation (23) may then be expressed as 
/n= 1 

M 

X' • £ [ (a m +Y m ) V m +y m g(X) ] . ( 24 ) 

/n«l 

Thus, with this limitation on the threshold value, the 
resultant unipolar positive binary-neuron vector will 
be the same when the value of X' in either Equation 
15 (23) or Equation (24) is used in taking the threshold. 

However, the addition operation in Equation (24) is 
easily realizable in optics, so it is that equation 
that is implemented in an inner-product unipolar TABAM 
architecture with adaptive thresholding which will be 
20 described below with reference to FIG. 8. It can be 
shown that from a theoretical point of view, the 
restriction placed on the threshold is reasonable. 
Nevertheless, the proper value of the threshold should 
be determined empirically by computer simulation and 
comparison with an optical experimentation. 


25 
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EXAMPLES AND EXTENSIONS OF A TAB AH 

In the following examples, the TABAM models are 
described and the adaptive threshold values of the 
TABAM are determined through theoretical derivations 
5 in the following discussion. 

As noted hereinbefore, it is difficult to imple- 
ment subtraction using optics. To avoid subtraction, 
an adaptive thresholding technique is used in the 
TABAM examples . Before the adaptive thresholding 
10 techniques are discussed, the TABAM model is briefly 
reviewed. One assumes that the ith component of the 
state vector x L at time t+1 may be written as a 
function of i (where t is an integer, a discrete 
representation of the time steps) as follows: 

^(t+l) = a m (t)+I if < 25a ) 

J77»l 

15 where 

I i = (25b) 

m-l 


6 m ( t) = exp 


-P£ [<7(^(0 ) -Uif 1 


(26c) 


V i- 1 


a m (t) = V Mjgixjit ) ) , 


(25d) 


where g(Xj(t)) is a logistic (sigmoidal) function 
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g(Xj (t) ) = 1/ (l+exp(-a[ Xj ( t)-O(t)])). (25e) 

The constant a in the sigmoidal, or nonlinear 
activation function decides the slope of the nonlinear 
thresholding function and 0(t) is the threshold. In 
Equation (1), u m i denotes the ith component of the mth 
stored vector V m , 0 is a constant, and M and N are the 
numbers of stored vectors and the number of neurons, 
respectively. Based on the property of the unipolar 
representation of a binary system, Equation (25a) can 
be rewritten as 

Xi ( t+1) -£ a m ( t)[(l+5“( t ) ) M m i -g{x 1 ( t) ) 5 m ( t) ], (26) 

/n-1 


since 

g{x i { t))-u"»l, 0, or -1 (27a) 

and 

[ffUj ( t) ) -uTJ 1/3 = gixtit H-iiJ. < 27b > 

Unipolar Inner-Product TABAM (UIT) Model 

Instead of implementing the subtraction in 
Equation (26) optically, an adaptive threshold func- 
tion is introduced following the modification of 
Equation (26) as follows: 

Xi ( t +1 ) =£ cc m ( t) [ ( 1 +5 m ( t) ) u i+g( Xi ( t) ) 6 m ( t) ] . ( 28 ) 

m-1 


Optical implementation of Equation (28) is referred to 
as the UIT. 

In Equation (28), a logistic function, as illus- 
trated in FIG. 7 with an adaptive threshold 0(t), is 
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used to perform the nonlinear transform such that the 
output attains quasi-binary states, which are defined 
when the constant a in the logistic function becomes 
very large but is still finite. The logistic function 
is continuous and has a large slope around the thresh- 
old 0(t). This will provide the terminal attractor 
with a continuous logistic function in a binary 
system. The selection of the constant a depends on 
the numerical resolution in the system. 

The adaptive threshold 0(t) will be determined 
for an ideal case and then generalized for a realistic 
case. In the ideal case, all of the stored states are 
orthogonal to one another; hence, the system is 
crosstalk-free . 

When there is no crosstalk, the threshold 6{t) is 
set as 


0(t) 


= a 


"'(t) 


+ 8 m (t) 


(29) 


where a m (t) is the inner product between x(t) and the 

mth stored vector, V m and 

o^(t) = 0 for all nt j* m'. 

Furthermore, the threshold 0(t) is set according 
to the following four possible cases: 

Case 1: If u" =1 , and g(x i (t)) - 1, then x i ft+l ) ■ a m (t)(l+2ff” (t)) 

Case 2: If U™=0, and g(x i (t)) - 0, then Xj.ft+1) - 0. 

Case 3: If U i =1 , and g(x i (t)) - 0, then x i (t+l) - a m (t) (1+0" (t)) 

case 4: If l) f=0, and g(x ± (t)) - I, then - a" (tfff" (t) . 

In order to cause the states to converge to the 
stored states, a threshold should be based on Equation 
(29) for maximum noise immunity between cases (3) and 
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(4) as shown in FIG. 8. The threshold is set between 
the lower value of the desired output ( state 1 ) and 
the higher value of the desired output (state 0). For 
a more practical situation, the stored states are not 
5 orthogonal to one another, so crosstalk often occurs 
between similar stored vectors. For the nonorthogonal 
case, 0(t) is set to be: 


M 


0(t) a m ( t) 


m»l 


+6 m (t) 


(30) 


Crosstalk Inner-Product Terminal-Attractor (CRIT) 
Model 

10 When the crosstalk corrupts the desired state due 

to the weighing process of the inner-product approach, 
and because every stored vector contributes the inner 
product to the summation in Equation (25) even though 
it is quite different from the input vector (as long 
15 as the inner product is not zero), the threshold 0(t) 
set by Equation (30) cannot provide an appropriate 
thresholding mechanism. In order to reduce/eliminate 
the crosstalk problem, a new model called CRIT is 
presented as an example as follows: 

x i ( t+1) =53 a m ( t) 5 m ( t)[(l+6"( C) )\)i+g(Xi ( t) ) 6 m ( t) ] . (31) 

m-l 


20 Equation (31) indicates that the crosstalk be- 

tween the nonorthogonal stored vectors is exponential- 
ly weighed and reduced by the exponential term 6 m (t). 
The property of the basin of terminal attractors is 
controlled by the value of j8 in 6 m (t), which will be 
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discussed below in detail. Using the CRIT example, 
the retrieved state vector is highly likely to be 
placed in a correct basin similar to the effect 
resulting from the multiplication between the weight 
matrix and the nonlinearly activated input vector m 
Zak's model. Then the corresponding terminal attract- 
or forces the state vector to converge rapidly to the 
bottom of the basin. This example does not need time 
for training and is more suitable for optoelectronic 
implementation for terminal attractors since the 
stored vectors can be used directly. The threshold 
can be set in a manner similar to that used in Equa- 
tion (30) as follows: 


M 


0(t) = £ a m (t)b m (t) 


70 -1 


- 1 + 6 * 


(t) 


( 32 ) 


Basic to the UIT and the CRIT is the terminal 
attractor, I if used in each iteration. Consequently, 
basic to this invention is a TABAM which, unlike that 
of Zak, supra, can be implemented optically to provide 
a neural dynamic system having finite relaxation 
times, no spurious states, and virtually infinite 
stability for use in real-time, high-density associa- 
tive memory applications. 


Computer Simulation 

In order to test the effectiveness of the exam- 
ples of UIT and CRIT using the adaptive threshold, a 
computer simulation was used. The XOR logic operation 
was used to detect the Hamming distance between the 
state vector, x(t ) , and the stored vectors, V 1 ". The 
input-output relationship of XOR operation is shown in 

Table 2. 
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Table 2. Exclusive OR (XOR) Relationship 


Input 1 

Input 2 

Output 

1 

1 

1 

0 

1 

0 

1 

0 

0 

0 

0 

1 


The Hamming distance between the state vector, x(t ) , 
and the stored vectors can be computed by using XOR as 
follows. For a unipolar system, 


[gU i (t))-x>tf*\g(x 1 (t))-» m i \ 


(33) 


Because g(x L (t) is very close to 0 or 1, and u m i is 0 
or 1, Equation (33) is a reasonable approximation. 
The Hamming distance between g(x i (t) and u m j_ is shown 
in Table 3 . 

Table 3. Hamming Distance Between gfxj^t)) and u™ 


g{x 1 ( t) ) 


|g(Xi(t) ) -u?| 

1 

1 

0 

0 

1 

1 

1 

0 

1 

0 

0 

0 


Based on Tables 2 and 3, the Hamming distance is 


found to be 
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AT 



i-l 


=N-a m { t) , 


(34) 


where 


cc m (t) - '£g(x 1 (t))XOR <. 

i-l 

Equation (34) provides a similar measure between 
g(x(t)) and V® and is equivalent to the inner product 
of a bipolar system. The exponential term can be 
5 calculated easily after the Hamming distance is 
obtained. 

6 m ( t) =S-P[w-«" ( t)] (35) 

based on the definition in Equation (25). 

Simulation Results 

Based on the above, two computer simulation 
10 programs were used to test the feasibility of the 
models: (1) an exhaustive test program for simulating 
every possible stored and input vectors in small-scale 
networks of UIT and CRIT, and (2) a Monte Carlo 
simulation code for testing randomly generated stored 
15 and input vectors in large-scale networks of CRIT. 

In the first program, the number of neurons and 
stored states selected is small. The number of stored 
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vectors is two and three for ail the possible states 
in both three-and four-neuron networks. For small N 
and M, all possible combinations of vectors are stored 
and retrieval is tested for all possible input vectors 
5 using an IBM PC 386 for the simulation. An associa- 
tive recall result is considered accurate when the 
Hamming distance between the input vector and the 
stored vector to which it converges is the smallest. 
The accuracy can be determined in all cases except 
10 those in which the input vector has a Hamming distance 
equal to two or more stored vectors. In this case, it 
is impossible to decide to which os the stored states 
the input should converge; however, because the Ham- 
ming distances are equal, the choice is inconsequen- 
15 tial. The simulation results of the UIT (Equation 
(28)) and CRIT (Equation (31)) are presented in Table 
4. 


Table 4. Computer Simulation Results of UIT and CRIT 


Model 

N 

M 

Convergence 

UIT 

3 

2 

100% 

UIT 

3 

3 

100% 

UIT 

4 

2 

100% 

UIT 

4 

3 

95.7% 

CRIT 

3 

2 

100% 

CRIT 

3 

3 

100% 

CRIT 

4 

2 

100% 

CRIT 

4 

3 

100% 


30 
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The 95.7% convergence accuracy of M= 3 and N= 4 of the 
UIT is due to the crosstalk effect. The inaccuracy is 
removed by the CRIT as shown in Table 4 . When M and 
N are further increased, the computer time for exhaus- 
5 tive testing simulation becomes considerably longer. 
Hence, a Monte Carlo simulation is used to test 
randomly generated patterns to evaluate the accuracy 
of retrieval. 

For the second program, a Monte Carlo simulation 
10 code was developed in a SunSPARC 2 workstation to 
simulate a CRIT network with a set of randomly gener- 
ated stored vectors (al of which are different) and to 
test the network with a set of randomly generated 
input vectors to measure the accuracy of correct 
15 convergence. As shown in Table 4, only 16 cases are 
selected from a significant number of performed tests 
to demonstrate the perfect convergence of the CRIT 
network. Each row in the table specifies the number 
of neurons in the network, the number of stored 
20 vectors in a set, the number of test vectors in a set, 
and the number of sets of stored vectors tested. 
Using the last row of Table 4 as an example: 64 

different sets of stored patterns are used for test- 
ing. Each set of stored patterns has 1024 patterns 
25 and is tested by a set of 256 test patterns. Each set 
of test patterns is generated randomly and indepen- 
dently for each set of stored patterns. Based on the 
Monte Carlo simulation results, the CRIT model works 
well — achieving 100% accuracy — for large-scale 
30 networks even when M=4W. 

Based on the results of the computer simulation, 
it was seen that even with M=AN, perfectly accurate 
convergence could be accomplished by CRIT . 
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Experimental Set-up 

In discussing the feasibility of optoelectronic 
implementation of the models , a typical example with 
four neurons and two stored states is described below. 

5 The experimental set-up is shown in FIG. 9. For sim- 
plicity, M was chosen to be two and N>M. Because of 
the small number of images stored, a multifocus 
hololens was not needed to replicate the input vector 
x(t) . The LCTVs from an Epson video projector were 
10 used as spatial light modulators SLMl and SLM2 . The 
input vector x(t) from a frame grabber of a computer 
PC was sent to SLMl, which is illuminated by a colli- 
mated beam from an Argon ion laser 10 via a spreading 
lens 11 and a collimating lens 12. The image is 

15 Fourier transformed by lens Ll and filtered by two pin 
holes in a filter plate F in order to pass the +1 and 
-1 diffraction orders of grid patterns of the LCTV 
SLMl. The +1 and -1 orders, once they have passed 
through the pin holes (filters), are inversely Fourier 
20 transformed by lens L2, and the XOR operations are 
performed with the two stored vectors in SLM2 from the 
computer PC. The signals resulting from XOR opera- 
tions are detected by a CCD camera 20 and sent to the 
computer PC to calculate the next state vector, 
25 x(t+l). The output x(t+l) is the input as a new state 

to SLMl for the dynamical system. This process is 
iterated until convergence is reached. The converged 
state is displayed on SLMl or on a monitor 30. 

Experimental Results 

A typical example associated with the experiment 
is presented below. Assume the input vector is: 


30 
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1 0 
1 1 

where 1 represented a bright pixel and zero represents 
a dark pixel. The two stored images (vectors) are 
assumed to be: 


1 1 
1 1 


and 


1 1 
0 1 


denoted as stored vector No. 1 and No. 2, respective- 
ly. The output images resulting from the XOR opera- 
tions between the input and the two stored images are: 

1 0 
1 1 


and 


1 0 
0 1 


The output images following the XOR operations appear 
as two spots of different intensity on the CCD. The 
computer can use an adaptive thresholding technique as 
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shown in Equations (4) or (8) to perform the iterative 
computation. 

The result obtained for the UIT before taking the 
threshold is : 

7.7 6.4 

5.5 7.7 


5 and the threshold, 0, of the logistic function is 3.9 
(when one sets j8=10). After taking the threshold, the 
result is the stored vector No. 2, which shows an 
accurate retrieval. The result of the subsequent 
iteration before thresholding is: 

17 17 

13 17 

10 where 0=9. The result after thresholding is again the 
stored vector No. 2, indicating accurate convergence. 

On the other hand, the result obtained for the 
CRIT before thresholding is: 

2.3 1.8 

2.0 2.3 

After taking the threshold with 0=1.1 (/3=1.0), the 

15 result is the same as that of the UIT. The result 
after the next iteration before thresholding is: 

14 14 

12 14 
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hence converges to the correct state. The above 
example offers a feasibility demonstration of using a 
computer to perform nonlinear thresholding and feed- 
back. 

5 The inner-product neural network system shown in 

FIG. 10 will now be described to illustrate the 
optical implementation of a unipolar terminal-attrac- 
tor based on neural associative memory (TABAM) 
achieved through proper application of an adaptive 
10 thresholding technique described with reference to 
FIGs. 7 and 8 to avoid subtraction in the optics. 

The experimental set-up in FIG. 9 used to demon- 
strate the feasibility of the optical XOR operation is 
shown in the system of FIG. 10 and identified by the 
15 same reference characters used there to depict the 
inner-product (exclusive-OR) operation. The object of 
this example is to demonstrate the feasibility of 
optical implementation of a TABAM illustrated in FIG. 
1. For simplicity, M is chosen to be two while N is 
20 chosen to be sixteen. Because of the small number of 
images stored in a storage transparency S, a multifo- 
cus hololens is not needed to replicate the input 
vector. A photograph 13 of the input vector is used 
with a television camera 14 for entering the initial 
25 input vector x(t) through the electronically address- 
able spatial light modulator SLMl, e.g., a liquid 
crystal television spatial light modulator (LCTV SLM) 
while a switch 15 is in the position shown. 

The spatial light modulator SLMl is illuminated 
30 by a collimated beam from the argon laser 10 via the 
spreading lens 11, the collimating lens 12 and a beam- 
splitter 16. The image is Fourier transformed by the 
lens Ll and filtered by the two pin holes in the 
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filter plate F to pass the ±1 orders of the images 
diffracted by the inherent grid patterns of the 
spatial light modulator SLMl. The passed ±1 orders 
are refocused by a subsequent Fourier transform lens 
5 L2 and multiplied by two images stored in a transpar- 
ency 17 placed at a first multiplication plane. This 
completes the inner-product operation of step 1 . 

In practice, this example would not be feasible 
for a large number M of stored vectors. If M>2 , a 
10 Dammann grating would be used as the means for repli- 
cating the input vector x(t) a number of times equal 
to M, and in the usual application of the invention 
electronically addressable spatial light modulators 
would replace the transparency 17 which stored the M 
15 vectors at the first multiplication plane and a 
similar transparency 18 at a second multiplication 
plane needed in the next step. 

A fine grain diffuser 19 is used to convert the 
inner products into scalars (light intensity spots) 
20 which are focused by a Fourier transform lens L3 from 
the plane where the diffuser 19 is placed into the 
transparency 18 at the second multiplication plane 
referred to above. The focused light spots through 
the lens L3 represent the scalar quantity a m (t) of the 
25 inner products between the input vector x(t) and the 

stored vectors V* • The function of the diffuser is to 

create uniform light distributions that are propor- 
tional to the inner-product scalar. The uniformly 
distributed scalar light patterns at the second multi- 

30 plication plane are thus weighted by the vectors Vj” 

that are stored in the transparency 18 placed at the 
second multiplication plane. That completes step 2. 
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The weighted vectors u> m (t) of Equation (25a) are 

imaged by a Fourier transform lens L4 onto the CCD 
camera 20 the video output of which is transmitted to 
an electronically controlled liquid crystal spatial 
5 light modulator SLM2 . The vectors are there illumi- 
nated by coherent light from the laser 10 vis the 
lenses 11, 12 and the beamsplitter 17. For the 
purpose of improving the contrast of the liquid 
crystal spatial light modulator SLM2, the vector 
10 images introduced there are enhanced by lenses 21 and 
22 and low-pass filtered by a pin-hole spatial filter 
23. However, it should be noted that if a high 
contrast liquid crystal spatial light modulator is 
used, this spatial filtering process for enhancement 
15 can be omitted. In this example, the contrast en- 
hanced images are then reflected by separate mirrors 
24 and 25 that are oriented so that the images are 
superimposed on an output CCD 26 which serves the dual 
function of optical summation of the vector images at 
20 the input side (step 3) and transmitting the sum at 
the output side to a programmed digital computer PC 
for performing the operation Xj(fc+) of Equation (25a) 
by performing the calculations of Equations (25b), 
(25c) and (25d) . In each computation equation 
25 g(xj( t) ) is a logistic (sigmoidal) function set forth 

in Equation (25e), which includes the thresholding 
function 0(t). This concludes in step 4 a complete 
iteration. The next step 5 is to introduce the next 
input vector Xj(t+1) computed by the computer PC to 

30 the input spatial light modulator SLM1 via the ????? 
15 which, after the first vector x(t; is entered, is 
switched to its alternate state to receive each 
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subsequent x i (t+l) . Iterations of the five steps con- 
tinue until convergence is reached. The output from 
the PC is g(x 1 (t+ 1) ) . It should be noted that this 

optical implementation may be generated in accordance 
5 with the Equations (26) through (29) for a unipolar 
inner-product TABAM (UIT) model and in accordance with 
Equations (30) through (32) for a crosstalk inner- 
product terminal attractor (CRIT) . 

Although particular embodiments of the invention 
10 have been described and illustrated herein, it is 
recognized that modifications and variations may 
readily occur to those skilled in the art. Conse- 
quently, it is intended that the claims be interpreted 
to cover such modifications and equivalents. 
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UNIPOLAR TERMINAL- ATTRACTOR BASED NEURAL 
ASSOCIATIVE MEMORY WITH ADAPTIVE THRESHOLD 

ABSTRACT OF THE DISCLOSURE 

A unipolar terminal-attractor based neural 
associative memory (TABAM) system with adaptive 
threshold for perfect convergence is presented. By 
adaptively setting the threshold values for the 
dynamic iteration for the unipolar binary neuron 
states with terminal-attractors for the purpose of 
reducing the spurious states in a Hopfield neural 
network for associative memory and using the inner- 
product approach, perfect convergence and correct 
retrieval is achieved. Simulation is completed with 
a small number of stored states (M) and a small number 
of neurons ( N ) but a large M/N ratio. An experiment 
with optical exclusive-OR logic operation using LCTV 
SLMs shows the feasibility of optoelectronic implemen- 
tation of the models. A complete inner-product TABAM 
is implemented using a PC for calculation of adaptive 
threshold values to achieve a unipolar TABAM (UIT) in 
the case where there is no crosstalk, and a crosstalk 
model (CRIT) in the case where crosstalk corrupts the 
desired state. 
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